{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cea11cae",
   "metadata": {},
   "source": [
    "# Tool Success Evaluator\n",
    "\n",
    "### Getting Started\n",
    "This sample demonstrates how to use tool success evaluator on agent data. The supported input formats include:\n",
    "- simple data such as strings and `dict` describing agent responses;\n",
    "- user-agent conversations in the form of list of agent messages. \n",
    "\n",
    "Before you begin:\n",
    "\n",
    "```bash\n",
    "pip install azure-ai-evaluation\n",
    "```\n",
    "Set these environment variables with your own values:\n",
    "1) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
    "2) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
    "3) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
    "4) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49404a44",
   "metadata": {},
   "source": [
    "The Tool Success evaluator determines whether tool calls done by an AI agent includes failures or not.\n",
    "\n",
    "This evaluator focuses solely on tool call results and tool definitions, disregarding user's query to the agent, conversation history and agent's final response. Although tool definitions is optional, providing them can help the evaluator better understand the context of the tool calls made by the agent. Please note that this evaluator validates tool calls for potential technical failures like errors, exceptions, timeouts and empty results (only in cases where empty results could indicate a failure). It does not assess the correctness or the tool result itself, like mathematical errors and unrealistic field values like name=\"668656\".\n",
    "\n",
    "Scoring is binary:\n",
    "- TRUE: All tool calls were successful\n",
    "- FALSE: At least one tool call failed\n",
    "\n",
    "This evaluation focuses on measuring the technical success of tool execution, not the semantic correctness of the results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d91cb876",
   "metadata": {},
   "source": [
    "Tool Success requires following input:\n",
    "- Response - Response from Agent (or any GenAI App). This can be a single text response or a list of messages generated as part of Agent Response. The evaluator examines tool call results within the response.\n",
    "- Tool Definitions - (Optional) Tool(s) definition used by Agent. Providing tool definitions helps the evaluator better understand the context of the tool calls made by the agent.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e07d293d",
   "metadata": {},
   "source": [
    "### Initialize Tool Success Evaluator\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "919d1b9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azure.ai.evaluation._evaluators._tool_success import _ToolSuccessEvaluator\n",
    "from azure.ai.evaluation import AzureOpenAIModelConfiguration\n",
    "from pprint import pprint\n",
    "\n",
    "model_config = AzureOpenAIModelConfiguration(\n",
    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
    ")\n",
    "\n",
    "\n",
    "tool_call_success = _ToolSuccessEvaluator(model_config=model_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4739878c",
   "metadata": {},
   "source": [
    "### Samples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2352536d",
   "metadata": {},
   "source": [
    "#### Evaluating Successful Tool Calls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ddc2e22",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Successful tool execution\n",
    "successful_response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_weather_123\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"Seattle\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"tool_call_id\": \"call_weather_123\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"15°C\", \"condition\": \"partly cloudy\", \"humidity\": \"68%\", \"wind\": \"8 mph NW\"}}],\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"The current weather in Seattle is partly cloudy with a temperature of 15°C.\",\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "result = tool_call_success(response=successful_response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c146de06",
   "metadata": {},
   "source": [
    "#### Response as String (str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "575d33eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Response as a simple string containing tool call information\n",
    "# This format is less common but still valid for the evaluator\n",
    "response_str = \"\"\"Tool call executed: fetch_weather(location=\"Paris\")\n",
    "Tool result: {\"temperature\": \"18°C\", \"condition\": \"clear sky\", \"humidity\": \"55%\"}\n",
    "The weather in Paris is currently clear with a temperature of 18°C.\"\"\"\n",
    "\n",
    "tool_definition = {\n",
    "    \"name\": \"fetch_weather\",\n",
    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
    "    \"parameters\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "    },\n",
    "}\n",
    "\n",
    "result = tool_call_success(response=response_str, tool_definitions=tool_definition)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7217ddc",
   "metadata": {},
   "source": [
    "#### Tool Definition as Single Dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f89ca010",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Successful tool execution with single tool definition dict\n",
    "successful_response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_nyc_weather\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"New York\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"tool_call_id\": \"call_nyc_weather\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"22°C\", \"condition\": \"sunny\", \"humidity\": \"45%\"}}],\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"The weather in New York is currently sunny with a temperature of 22°C.\",\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_definition_dict = {\n",
    "    \"name\": \"fetch_weather\",\n",
    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
    "    \"parameters\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "    },\n",
    "}\n",
    "\n",
    "result = tool_call_success(response=successful_response, tool_definitions=tool_definition_dict)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1aeb8c11",
   "metadata": {},
   "source": [
    "#### Multiple Successful Tool Calls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "444ee59c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Multiple successful tool executions\n",
    "multiple_successful_response = [\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:35Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"Seattle\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:37Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"14°C\", \"condition\": \"rainy\", \"precipitation\": \"light rain\"}}],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:38Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
    "                \"name\": \"send_email\",\n",
    "                \"arguments\": {\n",
    "                    \"recipient\": \"your_email@example.com\",\n",
    "                    \"subject\": \"Weather Information for Seattle\",\n",
    "                    \"body\": \"Current weather: 14°C and rainy with light rain.\",\n",
    "                },\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:41Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [\n",
    "            {\"type\": \"tool_result\", \"tool_result\": {\"message\": \"Email successfully sent to your_email@example.com.\", \"status\": \"delivered\"}}\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:42Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"I have successfully sent you an email with the weather information for Seattle.\",\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "result = tool_call_success(response=multiple_successful_response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63e43736",
   "metadata": {},
   "source": [
    "#### Tool Call Success without Tool Definitions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0ca062e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Successful execution without providing tool definitions\n",
    "simple_successful_response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_calendar_123\",\n",
    "                \"name\": \"get_calendar\",\n",
    "                \"arguments\": {\"start_date\": \"2025-01-01\", \"end_date\": \"2025-01-31\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"tool_call_id\": \"call_calendar_123\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"events\": [{\"date\": \"2025-01-15\", \"title\": \"Team Meeting\"}, {\"date\": \"2025-01-20\", \"title\": \"Project Review\"}]}}],\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"I found 2 events in your calendar for January: a Team Meeting on the 15th and a Project Review on the 20th.\",\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "# Evaluation without tool definitions\n",
    "result = tool_call_success(response=simple_successful_response)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97c1af0f",
   "metadata": {},
   "source": [
    "#### Example of Failed Tool Call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cafdee8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Failed tool execution with error\n",
    "failed_response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_weather_456\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"InvalidCity\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"tool_call_id\": \"call_weather_456\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"error\": \"Location not found\", \"status\": \"failed\", \"code\": 404}}],\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"I'm sorry, I couldn't retrieve the weather information due to an error.\",\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "# This should score FALSE due to tool failure\n",
    "result = tool_call_success(response=failed_response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_agent_evaluator_prp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
